1. Field of the Invention
The invention relates generally to the transfer of data within computing systems and more particularly to the transfer of data between different time domains using multi-lane communication links.
2. Background of the Invention
Referring to FIG. 1, a point-to-point communications link consists of three basic elements: a transmitter 10; a communications medium 12; and a receiver 14. Generally speaking, transmitter 10 transmits data using a local clock signal. Receiver 14 likewise uses a local clock signal to receive the data, but this clock signal is local to the receiver, and not the transmitter. Put another way, the clock signals used by transmitter 10 and receiver 14 are generated from different sources and are not identical. Transmitter 10 and receiver 14 can therefore be said to reside in different clock domains. (Each clock domain comprises the portion of the system in which the respective clock signal is used as the basis for transferring data or for performing its respective operations.)
While the clock signals in each clock domain may be designed to have the same frequency, it is virtually impossible to generate identical clock signals in any two clock domains. Typically, there is a phase mismatch and a slight frequency mismatch between the clock signals. The phase difference between the two signals can easily be handled by inserting a buffer (sometimes referred to as an elastic buffer) in the communications link so that data is written to the buffer according to the clock signal of the transmitting domain and retrieved from the buffer according to clock signal of the receiving domain. The frequency mismatch between the clock signals is somewhat more difficult to deal with. Because the rate at which data is transmitted over the communications link is not the same as the rate at which data can be received from the communications link, overflow or underflow conditions can result. These conditions can, in turn, cause corruption of the data and failure of the attempted transfer. Even if a buffer is used in the communications link, the difference in the amounts of data transmitted and received is cumulative, and the buffer can eventually overflow or underflow.
The problem of buffer overflow/underflow in a single communications channel can be resolved through the use of fill words which are added to or deleted from an elastic buffer. Fill words are a set of predefined symbols which occupy positions in the data stream, but which do not carry any of the information being transferred (e.g., the IDLE ordered set in the Fibre Channel protocol.) As the amount of data in the elastic buffer increases and the buffer approaches an overflow condition, fill words can be deleted from the output data stream. This effectively allows the output to catch up with the input. As the amount of data in the elastic buffer decreases and the buffer approaches an underflow condition, fill words can be added to the output data stream. This effectively allows slows down the output so that the input can catch up with it.
The problem of buffer overflow/underflow in a multilane communications link, however, cannot be resolved simply by implementing this same solution in each of the lanes of the link. Because there are phase differences between the clock signals which drive the data transfers in the different lanes, fill words are likely to be added to or deleted from some, but not all of the lanes in a single clock period. As a result, the data transferred over one or more of the lanes would be more than one clock cycle out of phase with the remainder of the lanes. Consequently, the data (which was originally multiplexed from a single serial stream to a serial stream for each of the lanes) would be reassembled into a serial stream in the wrong order, corrupting the data stream.
One or more of the problems outlined above may be solved by the various embodiments of the invention. Broadly speaking, the invention comprises systems and methods for enabling data transfers over communications links having a plurality of lanes. The various embodiments of the invention generally involve the monitoring of elastic buffers corresponding to the lanes of the communications link to detect approaching overflow and underflow conditions. In response to detecting one of these conditions in any one of the buffers, fill words are added or deleted to the data streams in each of the lanes at approximately the same time to equalize the data input to and output from the buffers and avoid overflow or underflow of the buffers.
In one embodiment, a system comprises a plurality of elastic buffers, each of which is coupled to one of a plurality of lanes in a communications link, and each of which is coupled to a buffer controller. Each of the elastic buffers receives data according to a clock signal derived from a first clock domain. The clock signal for each lane may be out of phase with those of the other lanes. This is due to skew which is introduced between the different clock signals for each lane by the communications link. Data is clocked out of the elastic buffer by a clock signal in a second clock domain. If there is a mismatch between the frequencies of the clock signals in the two clock domains, the amount of data in the elastic buffer will increase or decrease, causing an overflow or underflow condition. The buffer controller is configured to monitor each of the elastic buffers and to detect impending underflow or overflow conditions. In response to detecting one of these conditions, the buffer controller will cause the fill words to be added or deleted, respectively, to all of the elastic buffers, rather than only the buffer in which the overflow/underflow condition was detected.
In another embodiment, a method comprises providing an elastic buffer in the each lane of a multi-lane communications link, monitoring the amount of data in each of the elastic buffers, adding a fill word in each of the elastic buffers in response to detecting that the amount of data in any one of the buffers has reached a predetermined low threshold, and deleting a fill word in each of the elastic buffers in response to detecting that the amount of data in any one of the buffers has reached a predetermined high threshold. Some embodiments of the method may include monitoring one or more of the data streams to identify conditions in which addition or deletion of fill words might be delayed in one or more of the transmission lanes, or might violate constraints of the applicable data transmission protocol.